Skip to content

fix(ops): fail sidecar health waits fast when Docker is lost - #253

Merged
jsugg merged 1 commit into
mainfrom
fix/ci-health-wait-runtime-loss
Nov 11, 2026
Merged

jsugg merged 1 commit into
mainfrom
fix/ci-health-wait-runtime-loss

Conversation

@jsugg

@jsugg jsugg commented Nov 11, 2026 •

Copy link
Copy Markdown
Owner

Summary

Follow-up to #252. A Docker backend that dies inside a sidecars, channels or browser-lab health wait now ends that wait within seconds instead of polling out its timeout, so the #252 runtime-loss recovery starts right away.

Problem

#252 left this gap open: "a VM dying inside the 300s health check would burn that wait before recovery starts". Both health-wait layers polled docker compose ps blindly. On a failure they slept 2s and tried again until the deadline:

Wait Where Hosted macOS timeout
_wait_for_compose_service (inside clawops ops sidecars up) src/clawops/strongclaw_ops.py Postgres 180s (x2 via recreate), LiteLLM 300s, Qdrant 180s, Neo4j 600s
verify_sidecar_services_running / verify_compose_services_running tests/utils/helpers/_fresh_host/shell.py sidecars 300s, browser-lab 20s

The CI wait had a second defect. Each poll's subprocess timeout was the whole wait, and a hung docker compose ps raised subprocess.TimeoutExpired. That is not a FreshHostError, so the #252 guard could not catch it.

Design

Fail-fast in the product, shared with CI. A user running clawops ops sidecars up against a dead backend hits the same 300-600s wait, so the fix lives in clawops.strongclaw_runtime, next to ensure_docker_backend_ready:

  • probe_docker_backend(env, cwd) checks the DOCKER_HOST unix socket first. A missing socket is definitive. Otherwise it runs docker info with a 15s timeout.
  • DockerBackendWatch is called only after a status poll fails. It confirms loss on a definitive probe, or after 3 failed docker info probes in a row. Any successful poll or probe resets the streak.
  • A healthy backend with a slow container never probes, so it waits exactly as before. A failing compose ps while docker info answers also keeps waiting and still times out with the old message.

Classification. #252 decides runtime loss with a live probe, not by matching error text. The fail-fast only needs to end the step while the backend is still down. A unit test runs the real CI wait inside run_with_runtime_recovery and checks the whole path: the wait fails after 4s of fake time against a 300s timeout, the report says docker_socket_eof, the stack recovers, and the single retry passes.

User-facing change (clawops ops sidecars up)

  • A backend lost mid-wait now fails within one poll if the socket is gone, or about 3 polls (~6s plus probe time) otherwise. Before, it waited up to 180/300/600s.
  • Error: Docker backend became unreachable while waiting for compose service '<svc>' to reach running/healthy: <detail>; last status inspection error: <...>. Restart the Docker runtime, then rerun the operation.
  • DockerBackendLostError subclasses CommandError, so the exit code (1) and the --json payload (ok: false, error, failedService, command.exitCode: 1) keep their shape.
  • New structured event clawops.ops.sidecars.wait.backend_lost (service, target, detail, elapsed_ms), documented in OBSERVABILITY.md.
  • A lost backend during the first Postgres wait no longer triggers the pointless --force-recreate postgres step.
  • Unchanged: timeouts, the timeout message, and behavior while the backend answers.

Fault drill

FRESH_HOST_FAULT_INJECTION=orbstack-stop-during-sidecars-health arms a watcher thread around sidecars up. On two consecutive 1s sightings of the full sidecars stack running with a required service not yet healthy, which is the product health wait, it runs orb stop once. The watcher is joined before the guard's recovery runs, so recovery never races an in-flight orb stop. It is gated like the #252 drill: a dispatch-only input on Acceptance Tests, the core env expression, and the helper re-check. A contract test checks that every known fault is inert on push to main.

Fault-injection proof

Run: https://github.com/jsugg/strongclaw/actions/runs/42996741189, dispatched on this branch with fault_injection=orbstack-stop-during-sidecars-health. It is green on attempt 2.

macOS Fresh Host Sidecars (hypermemory), attempt 1:

10:03:23.189 [fresh-host] Fault injection orbstack-stop-during-sidecars-health: armed; watching for the sidecars health wait.
10:05:02.741 [fresh-host] Fault injection orbstack-stop-during-sidecars-health: stopping OrbStack (inside the sidecars health wait: litellm=starting, qdrant=starting).
10:05:12.273 [fresh-host] Command exited with 1 (attempt 1/2); reclaiming resources and retrying: ... sidecars up --repo-local-state
10:05:13.877 clawops.strongclaw_runtime.DockerBackendLostError: Docker backend became unreachable while waiting for compose service 'litellm' to reach running/healthy: Docker socket /Users/runner/.orbstack/run/docker.sock is missing; ...
10:05:26.398 [fresh-host] exercise-sidecars failed (Docker backend is not reachable from this shell.); probing the Docker runtime for loss.
10:05:26.402 [hosted-docker] Docker runtime lost during exercise-sidecars: orbstack_socket_missing.
10:05:26.403 [hosted-docker] OrbStack recovery attempt 1/2 triggered by orbstack_socket_missing; 180s slice.
10:05:40.913 [hosted-docker] Running: orb start
10:06:29.345 [hosted-docker] Docker ready after 1 probe(s).
10:06:58.929 [fresh-host] Docker runtime recovered; retrying exercise-sidecars once.
10:14:00.969 [fresh-host] Starting phase=exercise-channels-runtime.
10:14:16.562 [fresh-host] Starting phase=exercise-recovery-smoke.

macOS Fresh Host Sidecars (openclaw-default), attempt 2 (only that job was rerun, with the same inputs):

11:04:52.821 [fresh-host] Fault injection orbstack-stop-during-sidecars-health: stopping OrbStack (inside the sidecars health wait: litellm=starting, qdrant=starting).
11:05:08.843 clawops.strongclaw_runtime.DockerBackendLostError: Docker backend became unreachable while waiting for compose service 'litellm' to reach running/healthy: Docker socket /Users/runner/.orbstack/run/docker.sock is missing; ...
11:05:08.846 [fresh-host] Command exited with 1 (attempt 1/2); reclaiming resources and retrying: ... sidecars up --repo-local-state
11:05:18.481 [hosted-docker] Docker runtime lost during exercise-sidecars: orbstack_socket_missing.
11:05:31.299 [hosted-docker] Running: orb start
11:06:17.048 [hosted-docker] Docker ready after 1 probe(s).
11:06:41.976 [fresh-host] Docker runtime recovered; retrying exercise-sidecars once.
11:12:20.701 [fresh-host] Starting phase=exercise-channels-runtime.
11:12:35.900 [fresh-host] Starting phase=exercise-recovery-smoke.

The health wait now ends 10-16s after orb stop starts, and most of that is orb stop itself. Before, the LiteLLM wait would have polled for up to 300s.

Attempt 1 of the openclaw-default lane failed, and the failure was in the budget, not the fail-fast. The runner was unusually slow: setup took 14 min against about 6 min on the other lane, and the LiteLLM schema bootstrap timed out 4 times. The fault therefore fired 25 min into the 2100s scenario window. The fail-fast still worked: orb stop ran at 10:35:03 and DockerBackendLostError was raised at 10:35:29. With only about 549s left, below the 600s SCENARIO_RETRY_RESERVE_SECONDS, the #252 guard correctly refused with budget_exhausted: runtime_loss recovery ran out of time. Without this change the job would have failed the same way, just later.

Normal PR CI (switch off): https://github.com/jsugg/strongclaw/actions/runs/42996743683, all green. The Sidecars job shows FRESH_HOST_FAULT_INJECTION: (empty) and no injection or loss lines.

Tests

  • test_strongclaw_runtime.py: socket-missing probe without running docker, docker info ok/failed/timeout classification, watch streak/reset/definitive/validation.
  • test_strongclaw_ops.py (fake clock): socket loss fails with 0 sleeps; 3 failed probes fail at 4s vs a 300s timeout; a slow LiteLLM on a healthy backend waits 100 polls with 0 probes; a failing compose ps with an answering backend still times out with the old message; no Postgres recreate after loss; --json payload.
  • test_fresh_host.py (fake clock): the same matrix for the CI wait, plus a hung compose ps treated as a failed poll capped at 30s.
  • test_fresh_host_faults.py: health-pending detection, watcher fires on the second sighting, never fires after stop, inert unless requested, one-shot, and "did not fire" reporting.
  • test_fresh_host_runtime_loss.py: health-wait fail-fast into recovery plus one retry.
  • Contract: test_fault_injection_cannot_be_enabled_on_push_to_main covers every fault.

Local gates (raw)

make lint (ruff: all checks passed), make imports, make typecheck (mypy: no issues in 378 files; pyright: 0 errors), make test-unit (1253 passed), make test-contracts (150 passed), black --check (380 files unchanged). The commit hooks also ran actionlint, attribution and conventional-commit checks, and all passed.

A Docker backend that died inside a sidecars health wait was polled for
the rest of that wait (LiteLLM 300s, Neo4j 600s, Postgres 180s twice on
hosted macOS) before the #252 runtime-loss recovery could start.

Each failed `docker compose ps` inside a health wait now probes the
backend. A missing DOCKER_HOST unix socket, or `docker info` failing
three probes in a row, ends the wait at once; a healthy backend with a
slow container keeps waiting exactly as before.

- clawops: `_wait_for_compose_service` raises DockerBackendLostError, a
  CommandError subclass, so exit codes and the --json payload shape are
  unchanged, and emits clawops.ops.sidecars.wait.backend_lost.
  `sidecars up` no longer recreates Postgres after the backend is lost.
- CI: verify_compose_services_running (sidecars, channels retry,
  browser-lab) fails fast the same way. A hung compose ps is now a
  failed poll capped at 30s, not an uncaught TimeoutExpired that the
  runtime-loss guard could not catch.
- Drill: FRESH_HOST_FAULT_INJECTION=orbstack-stop-during-sidecars-health
  runs `orb stop` once while `sidecars up` waits on sidecar health. It is
  gated like the existing drill: manual dispatch off main only.
@jsugg
jsugg merged commit f13d7f2 into main Nov 11, 2026
36 of 37 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant